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3  Introduction 


Increasing  the  enrollment  of  patients  in  clinical  trials  is  important  to  making  progress  towards 
finding  more  effective  treatments  for  breast  cancer.  Accrual  is  complicated  by  a  large  number 
potential  studies  and  the  cost  and  complexity  of  determining  whether  a  patient  meets  the  nec¬ 
essary  eligibility  criteria.  Under  this  proposal,  we  are  developing  a  Web  based  expert  system 
which  can  determine  the  patients  eligibility  for  clinical  trials.  The  expert  system  is  designed 
to  take  into  account  the  cost  of  tests  which  are  required  to  meet  inclusion  criteria  and  acquire 
information  in  the  most  cost-effective  way  possible. 

Additionally,  it  is  important  to  be  able  to  easily  add  and  remove  clinical  trials  to  the  system. 
Trials  are  continually  becoming  available,  going  on  suspension  or  being  closed  to  accrual.  Towards 
this  end,  we  have  developed  a  companion  Web  based  system  that  enables  anyone  to  simply  enter 
the  information  required  to  describe  the  eligibility/ineligibility  criteria  for  a  clinical  trial.  A 
newly  entered  trial/protocol  can  then  be  directly  included  in  the  Clinical  trial  assignment  expert 
system  with  no  expert  intervention. 

Finally,  we  have  worked  on  methods  of  utilizing  probabilities  to  order  questions  so  that  those 
most  likely  to  rule  patient  out  of  a  protocol  are  first.  Recent  testing  has  shown  this  is  effective. 


4  Body 

In  this  third-year,  we  have  refined  the  original  prototype  to  produce  version  1.4.  We  have  tested  it 
with  data  from  187  retrospective  patients  and  approximately  200  more  recent  patients  including 
some  who  are  currently  undergoing  treatment.  We  have  extensively  tested  its  ability  to  order 
questions  associated  with  tests  to  save  dollar  costs  on  over  300  patients.  Table  1  summarizes 
our  matching  results  on  the  past  and  current  patients.  Patients  are  only  evaluated  for  trials  that 
are  currently  enrolling  patients.  The  trial  status  can  change  when  a  trial  is  put  on  suspension, 
closed,  brought  off  suspension,  or  initiated.  It  can  be  seen  that  the  system  finds  all  matches  that 
correspond  to  trials  in  which  patients  have  been  enrolled.  For  the  current  169  patients  for  which 
extensive  tests  have  been  done,  we  found  160  new  matches  to  protocols!  This  is  quite  promising 
for  increasing  accrual. 

The  cost  savings  are  shown  in  Table  2.  We  show  the  mean  test  costs  with  and  without  the 
ordering  heuristics.  Six  clinical  trials  have  incurred  selection  costs;  the  heuristics  have  reduced 
the  costs  for  four  of  these  trials,  and  have  not  affected  the  costs  for  the  other  two  trials. 

There  are  now  14  protocols  available  in  the  system.  At  the  present  time,  all  breast  cancer 
protocols  at  the  Moffitt  Cancer  Center  which  are  accruing  at  least  two  patients  a  month  are 
available  through  our  system.  We  are  adding  two  more  protocols  that  just  opened  and  are 
accruing  patients.  Our  automated  clinical  trial  updating  system  continues  to  allow  us  to  easily 
add  trials  to  the  system  [1,  2]. 

We  have  created  a  question  ordering  system  that  uses  a  crude  probabilistic  heuristic.  As 
patients  are  tested  against  the  system  over  time,  we  can  keep  a  record  of  how  many  times  each 
question  causes  a  patient  to  be  classified  as  ineligible  for  a  protocol.  These  results  will  take  the 
form  of  X  out  of  y  times  that  a  question  was  asked  it  directly  caused  a  patient  to  be  determined 
ineligible  for  protocol  z.  The  value  (|),p  can  be  treated  as  the  probability  that  question  q  will 
cause  a  patient  to  be  declared  ineligible  for  protocol  p.  This  value  will  be  reasonably  reliable 
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Table  1:  Results  of  selecting  clinical  trials  for  the  187  past  patients  and  169  current  patients. 
We  give  the  number  of  trial  participants,  selected  by  both  the  system  and  Moffitt  clinicians,  and 
the  number  of  the  other  eligible  patients,  identified  by  the  system. 

(a)  Results  for  the  187  past  patients. 


Clinical 

Trial 

Parti¬ 

cipants 

Other 

Eligible 

10822 

10 

5 

10840 

0 

19 

11072 

48 

26 

11378 

4 

19 

11992 

5 

6 

12100 

8 

20 

12101 

20 

30 

(b)  Results  for  the  169  current  patients. 


Clinical 

Trial 

Parti¬ 

cipants 

Other 

Eligible 

11132 

4 

1 

11931 

2 

26 

11971 

4 

0 

12100 

0 

5 

12101 

11 

52 

12385 

0 

19 

12601 

0 

1 

12643 

16 

36 

12757 

1 

3 

12775 

23 

17 
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Table  2:  Cost  savings  by  test  reordering. 


(a)  Results  for  the  187  past  patients. 


Clinical 

Trial 

Mean  Cost 

W/0  Test 
Reordering 

With  Test 
Reordering 

10822 

$70 

$11 

10840 

$0 

$0 

11072 

$209 

$60 

11378 

$35 

$19 

11992 

$0 

$0 

12100 

$0 

$0 

12101 

$0 

$0 

(b)  Results  for  the  169  current  patients. 


Clinical 

Trial 

Mean  Cost 

W/0  Test 
Reordering 

With  Test 
Reordering 

11132 

$0 

$0 

11931 

$0 

$0 

11971 

$192 

$192 

12100 

$0 

$0 

12101 

$0 

$0 

12385 

$0 

$0 

12601 

$36 

$3 

12643 

$0 

$0 

12757 

$107 

$107 

12775 

$0 

$0 
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after  the  question  has  been  asked  more  than  30  times.  At  that  point,  it  can  be  used  to  reorder 
questions.  The  question  with  the  highest  probability  of  making  a  patient  ineligible  for  a  trial  can 
be  displayed  first.  By  doing  this  patients  will  be  quickly  determined  ineligible  with  a  minimum 
number  of  questions. 

Preliminary  experiments  have  been  done  which  indicate  this  approach  does  in  fact  reduce  the 
number  of  questions  necessary  to  determine  eligibility.  Results  are  shown  in  Table  3. 

We  selected  90  patients  at  random  from  our  list  of  patients  and  used  their  data  in  experiments. 
A  ten-fold  cross  validation  was  carried  out,  so  that  the  system  was  trained  on  81  patients  and 
the  remaining  9  patients  were  tested  using  the  system.  The  test  was  done  on  six  protocols  for 
which  90  patients  had  been  tested.  As  can  be  seen  in  Table  3,  the  probabilistic  system  allows 
approximately  13%  less  questions  to  be  answered  to  determine  eligibility,  on  average. 


Table  3:  Probabilistic  question  ordering  vs.  anal3d.ic  question  ordering. 


Ten-fold  cross  validation 

Average  number  of  questions 

Protocol 

Probabilistic 

Analytical 

Difference 

System 

System 

Difference 

% 

11931 

15.35 

18.90 

3.55 

18.78 

12100 

13.85 

13.95 

0.10 

0.72 

12101 

21.65 

24.75 

3.10 

12.53 

12521 

14.75 

19.05 

4.30 

22.57 

12601 

13.90 

15.70 

1.80 

11.46 

12777 

14.40 

16.10 

1.70 

10.56 

Average 

15.65 

18.08 

2.43 

13.42 

Key  Research  Accomplishments: 

•  We  have  enhanced  our  prototype  system  to  very  stable  version  1.4.  We  have  corrected  cost 
functionality  (mostly  by  getting  the  costs  of  tests  correct  and  determining  all  tests  that  are 
done  in  the  routine  care)  and  tested  this  successfully. 

•  Utilizing  retrospective  patient  data  and  cmrent  patient  data,  it  has  been  found  that  patients 
are  eligible  for  multiple  protocols/trials.  Further,  with  current  patient  data  we  find  patients 
eligible  for  trials  and  not  put  on  any  trial. 

•  Extensive  testing  of  cost  functionality  has  been  done.  We  have  determined  that  in  many 
cases  there  is  no  possibility  of  saving  costs.  However,  when  it  is  possible  the  cost  mechanism 
recommends  questions  in  order  that  will  always  allow  eligibility  to  be  determined  in  the 
minimal  cost  fashion. 

•  We  have  developed  a  method  of  determining  probabilities  that  questions  will  show  patients 
are  ineligible  for  trials.  The  probabilities  are  determined  empirically  while  the  system  is  in 
use.  We  showed  that  the  use  of  these  probabilities  to  order  questions  on  a  query  page  will 
result  in  the  need  to  answer  less  questions  at  all  times. 
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Reportable  Outcomes:  We  have  had  a  paper  [3],  which  is  attached,  accepted  to  the  2003 
IEEE  International  Conference  on  Systems,  Man,  and  Cybernetics.  We  have  a  revised  journal 
submission  that  got  reasonably  positive  reviews  undergoing  the  second  round  of  review.  A  web 
prototype  of  the  clinical  trial  assignment  system  is  available  at  http:/ /morden.csee.usf.edu/moffit 
with  password  available  from  the  principal  investigator. 


5  Conclusions 

We  have  developed  a  scalable  prototype  which  currently  can  determine  eligibility  for  fourteen 
breast  cancer  clinical  trials.  The  system  has  been  tested  using  retrospective  data  from  187 
patients  who  are  assigned  to  some  clinical  trial  and  more  recently  active  patients  numbering  169. 
The  system  correctly  finds  cases  in  which  a  patient  is  eligible  for  multiple  clinical  trials.  It  has 
found  160  matching  trials  for  the  169  current  patients.  This  indicates  that  it  is  quite  likely  they 
use  of  the  system  will  significantly  increase  accruals  to  trials. 

The  system  is  able  to  utilize  monetary  cost  in  requesting  tests  to  rule  in/rule  out  a  patient 
from  the  set  of  available  clinical  trials.  The  default  ordering  of  questions  allows  the  system  user 
to  rapidly  determine  the  eligibility  or  ineligibility  of  a  patient  for  any  subset  of  the  available 
clinical  trials  entered  into  the  system.  We  have  been  able  to  show  a  good  average  cost  saving  by 
using  the  cost  feature  to  order  questions.  Of  course,  there  is  no  guarantee  that  a  clinician  would 
order  tests  as  suggested  by  the  question  ordering  of  our  system.  However,  the  potential  for  cost 
savings  is  significant. 

The  system  is  Web  based  and  password  protected.  It  provides  rapid  response  when  a  person 
enters  answers  to  one  or  more  questions  on  a  page  of  system  selected  questions.  It  can  be  used 
from  any  computer  on  the  World  Wide  Web.  Hence,  community  physicians  will  be  able  to 
determine  the  potential  eligibility  (they  may  not  wish  to  run  all  tests)  of  the  patient  for  clinical 
trials  at  cancer  centers  in  their  region. 

A  prototype  to  enable  physicians,  nurses  or  technicians  to  enter  new  protocols  has  been 
completed.  The  system  is  now  in  use.  It  reduces  the  time  required  to  add  a  new  trial  or  protocol 
to  approximately  1  hour.  It  enables  non-computer  scientists  to  add  trial/protocols  to  the  system. 
This  knowledge  acquisition  tool  has  been  designed  to  minimize/eliminate  the  cases  where  similar 
questions  acquiring  essentially  the  same  information  would  have  to  be  asked.  This  feature  has 
the  potential  to  cause  slight  changes  to  the  wording  of  inclusion/exclusion  criteria.  We  believe 
that  this  change  is  minor  and  will  have  no  effect  on  IRB  approval. 

Last  year,  we  intended  to  evaluate  whether  IRB  approval  would  be  affected.  However,  institu¬ 
tional  issues  prevented  this.  However,  this  year  we  will  have  new  protocols  entered  using  existing 
questions  and  plan  to  go  back  to  the  IRB  board  to  discuss  any  changes  in  criteria  wording  to  fit 
existing  questions  within  the  system.  An  example  would  be  a  protocol  in  which  there  are  two 
questions  which  ask  “is  a  test  value  is  greater  than  some  threshold”  and  then  a  separate  question 
that  asks  if  it  is  less  than  some  threshold,  versus  a  single  question  which  asks  if  a  test  is  in  some 
range.  We  believe  that  such  a  change  is  trivial,  but  this  must  be  addressed  in  practice  and  we 
will  evaluate  whether  it  causes  review  board  decisions  to  potentially  change. 
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Abstract  —  When  clinicians  test  a  new  treatment  pro¬ 
cedure,  they  need  to  recruit  patients  with  appropriate 
medical  conditions.  We  have  developed  an  expert  system 
that  helps  clinicians  to  select  patients  for  experimen¬ 
tal  treatments,  and  to  reduce  the  cost  of  related  medical 
tests.  We  describe  experiments  on  selection  of  patients 
for  new  treatments  at  the  Moffitt  Cancer  Center.^ 

Keywords:  Medical  expert  systems,  breast  cancer, 
cost  reduction. 


1  Introduction 

When  clinicians  conduct  treatment  experiments, 
called  clinical  trials,  they  have  to  recruit  participants 
among  current  patients.  To  select  prospective  partici¬ 
pants,  clinicians  analyze  the  data  of  available  patients, 
which  has  traditionally  been  a  manual  process.  Studies 
have  shown  that  clinicians  miss  up  to  60%  of  the  eligible 
patients,  which  delays  the  completion  of  clinical  trials. 

To  address  this  problem,  researchers  built  several  sys¬ 
tems  that  helped  clinicians  to  select  trial  participants. 
Ohno-Machado  et  al.  [1993]  developed  the  AIDS^  system, 
which  selected  AIDS  patients  for  clinical  trials.  Musen 
et  al.  [1996]  built  a  rule-based  system,  called  EON,  that 
also  assigned  AIDS  patients  to  clinical  trials.  Bouaud 
et  al.  [1998;  2000]  created  the  ONCODOC  system,  which 
suggested  trials  for  cancer  patients.  Seroussi  et  al.  [1999; 
2001a;  2001b]  used  ONCODOC  to  select  trial  participants 
at  two  hospitals,  which  helped  to  increase  the  number 
of  selected  patients  by  a  factor  of  three. 

The  National  Cancer  Institute  has  created  a  search 
engine  for  selection  of  clinical  trials,  available  through 
the  Internet  at  www.cancer.gov/search/clinicaLtrials.  It 
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prompts  a  user  to  answer  several  questions  about  a  pa¬ 
tient,  and  gives  a  list  of  potentially  matching  trials;  how¬ 
ever,  it  does  not  determine  whether  the  patient  satisfies 
all  requirements  for  these  trials. 

Fallowfield  et  al.  [1997]  studied  how  physicians  se¬ 
lected  cancer  patients  for  clinical  trials,  and  compared 
manual  and  automated  selection.  They  showed  that  ex¬ 
pert  systems  could  improve  the  selection  accuracy,  but 
physicians  were  reluctant  to  use  these  systems.  Carlson 
et  al.  [1995]  conducted  similar  studies  with  AIDS  trials, 
and  also  concluded  that  expert  systems  could  lead  to  a 
more  accurate  selection. 

A  recent  project  at  the  University  of  South  Florida 
has  also  been  aimed  at  automated  identification  of 
prospective  trial  participants.  Theocharous  developed 
a  Bayesian  system  that  selected  clinical  trials  for  can¬ 
cer  patients  [Theocharous,  1996;  Papaconstantinou  et 
al,  1998],  and  Bhanja  et  al  [1998]  built  a  qualitative 
rule-based  system  for  the  same  task. 

We  have  continued  their  work,  built  a  new  ver¬ 
sion  of  the  rule-based  system  [Kokku  et  al,  2002; 
Nikiforou  et  al,  2002;  Nikiforou,  2002],  and  applied  it  to 
selection  of  patients  for  breast-cancer  trials  at  the  Mof¬ 
fitt  Cancer  Center,  located  on  campus  of  the  University 
of  South  Florida.  We  outline  the  design  of  this  system 
and  present  an  empirical  evaluation  of  its  effectiveness. 

2  Knowledge  base 

Physicians  at  the  Moffitt  Cancer  Center  conduct 
about  150  clinical  trials.  We  have  developed  an  expert 
system  that  helps  clinicians  to  select  trials  for  eligible 
patients;  it  consists  of  a  knowledge  base  and  web-based 
interface  for  entering  patient  data.  The  knowledge  base 
contains  information  about  related  medical  tests,  as  well 
as  logical  expressions  that  determine  a  patient’s  eligi¬ 
bility  for  each  trial.  The  description  of  a  medical  test 
includes  its  dollar  cost  and  list  of  questions  that  can 


(a)  TESTS  AND  QUESTIONS 

General  information 
What  is  the  patient’s  sex? 

What  is  the  patient’s  age? 

Mammogram,  Cost  is  $150 
What  is  the  cancer  stage? 

Does  the  patient  have  invasive  cancer? 

Biopsy,  Cost  is  $400 

How  many  lymph  nodes  have  tumor  cells? 

What  is  the  greatest  tumor  diameter? 

Electrocardiogram,  Cost  is  $200 

Does  the  patient  have  cardiac  arrhythmias? 

(b)  ELIGIBILITY  CRITERIA 

sex  =  FEMALE  and 
age  <  45  and 

cancer- stage  G  {ll,  III}  and 
invasive- cancer  =  NO  and 
lymph-nodes  <  3  and 
{arrhythmias  =  NO  or 
tumor- diameter  <  2.5) 

Figure  1:  Description  of  medical  tests  and  trial- 
eligibility  criteria  in  the  trial-selection  system. 

be  answered  based  on  the  test  results  (Figure  la).  The 
trial-eligibility  criteria  are  a  logical  expression,  which 
includes  variables  that  represent  the  patient  data,  as 
well  as  equalities,  inequalities,  ^‘set-element”  relations, 
conjunctions,  and  disjunctions  (Figure  lb). 

The  system  collects  data  until  it  can  determine 
whether  the  eligibility  expression  is  TRUE  or  FALSE.  For 
example,  if  a  clinician  uses  the  system  to  determine  a 
patient’s  eligibility  for  the  trial  in  Figure  1(b),  it  first 
asks  about  the  patient’s  sex  and  age.  If  the  patient  sat¬ 
isfies  the  corresponding  conditions,  the  system  asks  for 
the  mammogram  results,  and  then  requests  the  biopsy 
and  electrocardiogram  data.  The  ordering  of  tests  de¬ 
pends  on  their  costs  and  on  the  amount  of  information 
provided  by  test  results.  The  system  begins  with  the 
mammogram  because  it  is  cheaper  than  the  other  tests 
and  provides  data  for  two  clauses  of  the  eligibility  ex¬ 
pression. 

3  Selection  of  participants 

We  have  built  a  knowledge  base  for  the  breast- cancer 
trials  at  the  Moffitt  Cancer  Center,  including  five  com¬ 
pleted  trials  and  ten  current  trials.  We  have  applied 
the  system  to  retrospective  data  from  187  past  patients 
and  169  current  patients,  and  compared  the  results  with 
the  manual  selection  by  Moffitt  clinicians.  The  system 
has  identified  all  eligible  patients  for  each  trial,  whereas 
the  clinicians  have  selected  about  half  of  the  eligible  pa¬ 
tients.  We  summarize  the  results  for  the  past  patients 


in  Table  1(a),  and  the  results  for  the  current  patients 
in  Table  1(b).  The  “participants”  column  shows  the 
number  of  actual  participants  of  each  trial;  the  “other 
eligible”  column  gives  the  number  of  the  other  eligible 
patients  identified  by  the  system. 

For  every  current  patient  who  has  not  participated 
in  a  matching  trial,  we  have  checked  whether  she  par¬ 
ticipated  in  any  other  trial,  and  we  show  the  results  in 
Table  2.  We  have  not  done  a  similar  analysis  for  the  past 
patients  because  of  insufficient  data.  The  “incompati¬ 
ble”  column  in  Table  2  includes  the  number  of  eligible 
patients  who  skipped  a  specified  trial  because  of  partici¬ 
pation  in  another  incompatible  trial.  The  “compatible” 
column  shows  the  number  of  patients  who  participated 
in  another  compatible  trial,  and  could  also  have  partici¬ 
pated  in  the  specified  trial.  Finally,  the  “no  other  trial” 
column  gives  the  number  of  eligible  patients  who  have 
not  participated  in  any  trial. 

The  results  show  that  the  system  can  identify  eligible 
patients  who  have  not  been  selected  by  clinicians;  thus, 
it  can  increase  the  number  of  trial  participants.  For  the 
patients  in  the  reported  experiments,  it  could  increase 
the  number  of  participants  by  a  factor  of  two. 

4  Cost  reduction 

If  the  patient  records  do  not  provide  enough  data, 
clinicians  perform  medical  tests  as  part  of  the  selection 
process.  They  can  reduce  the  overall  test  cost  by  first 
requiring  inexpensive  tests,  and  then  using  their  results 
to  avoid  some  expensive  tests. 

The  developed  system  suggests  the  ordering  of  tests 
that  reduces  their  expected  cost.  After  getting  the  re¬ 
sults  of  the  first  test,  it  re-evaluates  the  need  for  the 
other  tests  and  revises  their  ordering.  The  choice  of  the 
first  test  is  based  on  three  criteria.  The  system  scores 
all  required  tests  according  to  these  criteria,  computes 
a  linear  combination  of  the  three  scores  for  every  test, 
and  chooses  the  test  with  the  highest  score. 

1.  Cost  of  a  test.  The  system  gives  preference  to  less 
expensive  tests. 

2.  Immediate  decision.  If  a  test  can  lead  to  immedi¬ 
ate  acceptance  or  rejection  of  the  trial,  the  system 
prefers  it  to  other  tests, 

3.  Number  of  related  clauses.  The  system  prefers  the 
tests  that  provide  data  for  the  largest  number  of 
clauses  in  the  eligibility  expression. 

The  system  disregards  the  costs  of  tests  performed  in 
the  normal  course  of  treatment,  and  accounts  only  for 
the  costs  related  to  the  trial  selection.  For  example,  if  a 
patient  needs  a  mammogram  regardless  of  clinical-trial 
participation,  the  system  views  it  as  a  zero-cost  test.  On 
the  other  hand,  if  the  only  purpose  of  the  biopsy  and 
electrocardiogram  is  to  select  clinical  trials,  the  system 
uses  heuristics  to  order  these  tests. 


Table  1:  Results  of  selecting  trials  for  187  past  patients 
and  169  current  patients.  We  give  the  number  of  trial 
participants,  selected  by  both  the  system  and  Moffitt 
clinicians,  and  the  number  of  the  other  eligible  patients, 
identified  by  the  system. 

(a)  Results  for  the  187  past  patients. 


Clinical 

Trial 

Parti¬ 

cipants 

Other 

Eligible 

10822 

10 

5 

10840 
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19 

11072 

48 

26 
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(b)  Results  for  the  169  current  patients. 


Clinical 

Trial 

Parti¬ 

cipants 

Other 

Eligible 

11132 

4 

1 

11931 

2 

26 

11971 

4 

0 

12100 

0 

5 

12101 

11 

52 

12385 

0 

19 

12601 

0 

1 

12643 

16 

36 

12757 

1 

3 

12775 

23 

17 

Table  2:  Participation  of  the  patients  who  skipped  a 
matching  trial  in  other  trials.  We  show  the  number  of 
patients  who  skipped  the  trial  because  of  participation 
in  another  incompatible  trial;  the  number  of  patients 
who  were  on  another  trial  compatible  with  the  skipped 
trial;  and  the  number  of  eligible  patients  who  were  not 
on  any  trial. 


Table  3:  Cost  savings  by  test  reordering. 


(a)  Results  for  the  187  past  patients. 


Clinical 

Trial 

Mean  Cost 

Without  Test 
Reordering 

With  Test 
Reordering 

10822 

$70 

$11 

10840 

$0 

$0 

11072 

$209 

$60 

11378 

$35 

$19 

11992 

$0 

$0 

12100 

$0 

$0 

12101 

$0 

$0 

(b)  Results  for  the  169  current  patients. 
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Trial 

Mean  Cost 

Without  Test 
Reordering 

With  Test 
Reordering 

11132 

$0 

$0 

11931 

$0 

$0 

11971 

$192 

$192 

12100 

$0 

$0 

12101 

$0 

$0 

12385 

$0 

$0 

12601 

$36 

$3 

12643 

$0 

$0 

12757 

$107 

$107 

12775 

$0 

$0 

$250 

$200 

J  $150 

1  $100 
s 

$50 
$0 

(a)  Results  for  the  187  past  patients. 
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(b)  Results  for  the  169  current  patients. 

Figure  2:  Costs  with  and  without  test  reordering.  We 
plot  the  results  for  the  six  trials  that  have  incurred 
nonzero  selection  costs. 
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We  show  the  mean  test  costs  with  and  without  the 
ordering  heuristics  in  Table  3,  and  give  a  graphical  view 
of  the  cost  savings  in  Figure  2.  The  results  confirm 
that  the  heuristics  reduce  the  cost  of  the  selection  pro¬ 
cess.  Six  clinical  trials  have  incurred  selection  costs;  the 
heuristics  have  reduced  the  costs  for  four  of  these  trials, 
and  have  not  affected  the  cost  for  the  other  two  trial. 

The  results  in  Table  3(a)  differ  from  similar  experi¬ 
ments  with  an  earlier  version  of  the  system,  reported 
in  the  article  by  Kokku  et  al  [2002],  because  of  two 
changes  in  the  system.  First,  the  current  version  disre¬ 
gards  the  costs  of  the  tests  required  for  the  regular  treat¬ 
ment,  which  do  not  affect  the  trial- selection  expenses, 
whereas  the  earlier  version  counted  all  costs.  Second, 
some  costs  in  the  old  system  were  out-of-date,  and  we 
have  corrected  them  based  on  the  data  from  the  Moffitt 
accounting  department. 

5  Concluding  remarks 

We  have  developed  an  expert  system  that  selects  clin¬ 
ical  trials  for  eligible  patients.  Experiments  have  con¬ 
firmed  that  the  system  has  the  potential  to  improve  the 
accuracy  of  selecting  trial  participants.  They  have  also 
shown  that  the  ordering  of  medical  tests  affects  their 
overall  cost,  and  the  implemented  heuristics  reduce  the 
cost  of  finding  trial  participants. 
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